posted 10-25-2006 08:40 AM         

[This message has been edited by stat (edited 10-25-2006).]

[This message has been edited by stat (edited 10-25-2006).]

[This message has been edited by stat (edited 10-25-2006).]

posted 10-25-2006 09:31 AM            

As for your point above, it is well taken. That is the problem with sex offender testing: we don't have much research to support it, and we don't know if the CQs are relevant (or more important). It essentially makes a CQT and R/I test. The Feds had a scoring algorithm for R/Is, but I don't know if it's done or if it's classified. Who knows if it'll be available to the rest of us any time soon. There are a number of ways to evaluate such tests, and they all have some shortcomings.

I ran a GQT as a screening exam the other day, and the guy hit on the "Have you lied to me in any way..." question, which almost earned him an NDI score. I interrogated on that issue, and he confessed that he had lied to one of the RQs he had done okay on. It's a real problem, and it's the reason I like the R/I test as a screening exam. Anyhow, I evaluate screening exams by scoring (on CQTs), and then globally as I would an R/I test, and then I go from there.

posted 10-25-2006 09:57 AM         

[This message has been edited by stat (edited 10-25-2006).]

posted 10-25-2006 01:06 PM         

-----I just knew that my sheet music stand would have other uses---haha
Also, the software (Israeli?) which times to the milisecond the onset of arousels might become very handy due to the probable lengthiness of arousels due to the span of statement.

I am going to run this format on my wife tommorrow using classic controls as relevants just to see the stimulus patterns----yuck it up guys--but my wife is very cool.

indulge me here people.

posted 10-26-2006 08:55 AM            

But my head is so full of non-parametrics and CQT scoring right now, that I can only bob my head up and down like some bobble-headed buffoon who'se bingeing on bagels before bedtime.

You've got a lot of very intersting ideas here.

In some ways we may have been holding our own profession back, by not finding an effective balance between procedural adherance and empirical curiosity about all the diffferent reasons that explain how the dad-gum thing works.

I have to continually remind therapists and POs that it is not simply lying that causes reactions, there a combination of psychological and physiological mechanisms that interact at the polygraph. Some of them are well understood.

In short we are fooling ourselves if we try to think we will ever know "everything" about what offenders have been up to. We can learn more, but we will never know "everything."

The issues surrounding interrogating on Comparison Qs is complex, and in part involves the long-term effects of that you-lied-but-you-passed experience as a matter of routine, every few months, for periods of 10 years to lifetime ('cause that's how long they are on probation in Colorado). It also involves these conversations with therapists - who are sometimes just cheerleaders with initials behind their names (oh, sorry, that's the drug company reps). I can't tell you how many times I've admonished offenders and therapists that no polygraph examiner had ever used the words "flying colors," after the word "passed," though we had probably often used the word "miserably" after the word "failed."

I have a couple of recent countermeasure tests that are interesting, along with a couple clean tests.

One guy I tested twice - DI/confession on touching a stripper, and NDI/reg the stripper's request for $100 before she called the bouncer. Interestingly, the same "dancer" was involved in a similar incident two weeks earlier, only that time she was "bit," not touched, and the test subject was an attorney attending a bachelor party just two days before a scheduled wedding (I never heard if the wedding happened). I have another subpoena on a divorce case, involving an attorney who was arrested for soliciting. (Ya, just can't take those attorneys anywhere.)

Aside from the complexities of psychological set and attention/response, whether CQ are "relevant" is an conversation or discussion that could use a lot more careful articulation - as in... what exactly does that mean, or what does it mean to say that they are relevant (more important). If the question about lying to a therapist is more important that a question about with whom the offender has engaged in sexual contact - it may be that it is simply an effective control. The client defines the relevant issues (target issues for investigation), based on what information they need to proceed with treatment intervention and risk management efforts. For us to attempt to define what the relevant or target issues are is like Safeway telling me what groceries I need, or McDonald's telling us about our nutritional needs. In a strictly empirical sense, our task is simply to determine whether their are significant differences in the way the subject responds to some questions - RQs or CQs (these-here scores are different that those-there scores - at some statistically significant level.)

Value judgements about the "clinical significance" are an important concern, but are not the same as our concerns about the empirical accuracy of our test results - i.e., which quesions capture the greatest portion of the subject's response potential.

In between arguments with Sprint's horrible tech support, I've been playing with data standardizations, and rank-transformations, and component weightings, along with normalcy plots and Levene tests to try to justify the use of parametric algorithms in computer scoring. I've also begun to make use of distribution-free stats like Wilcoxon-W, Mann-Whitney-U, and Kruskal-Wallis-H tests to try to score CQT data. Like hand-scoring, computer scoring is not rocket-science. I'm finding that it might be possible to use common parametric and non-parametric statistics to score CQT data.

I'm not convinced that it couldn't be done with RI tests too, but it would take some better definition of measurement paradigms, and a clearer understanding of the complex range of phenomena underlying observed responses.

Here is a one page (18k) .pdf of a spreasheet scored results of a test on yet-another attorney (this one a fixated pedophile). Idenifi called this maintenance test NDI, as do hand scores. He basically lives on house-arrest, so its not surprising. You can see the q-q plots with a lot of granularity in the data.
http://www.raymondnelson.us/qc/sample_sig_10-25-06.pdf

One thing I am finding is that running only three tests barely satisfies sample (N) size and continuity requirements for even non-parametrics (especially if we want to employ the convenience of normal approximations). The Mann-Whitney U test has tables of critical values for N as small as 3 - which is what we've got on a mixed issue test in which each RQ is asked three times and scored separately. But what happens if one of those is artifacted. For another example the Wilcoxon W test (sometimes called T) lacks the power to achieve any significance at .05 with N<5. On the other hand, five charts, provides enough sample iterations to satisfy a most non-parametric requirements and some normal approximations.

Here is a link to an example. This one is a DI maintenance polygraph. I got a confession on a pornography question at R8. Identifi called it DI - with the most intense negative score at R6. Look at the difference between the overall score and the parsed scores for the individual RQs. Underneath this one page summary, there is a bit of complex re-ranking to compute separate point estimates of each RQ. You can also see that the non-parametrics don't have the same power as the parametrics.
http://www.raymondnelson.us/qc/sample_sig_10-25-06_2.pdf

I know its not very scientific to just plug my own test data into the spreadsheet - but this is still in the proof-of-concept stage. I've always wanted to know more about what the heck goes on underneath computer scoring systems.

Anyway, this ends our math lesson for today.

There will be a pop-quiz tommorow.

Peace,

r

------------------
"Gentlemen, you can't fight in here, this is the war room."
--(from Dr. Strangelove, 1964)

posted 10-26-2006 01:17 PM            

try

user: polyguest
password: torquemada - like the spanish inquisitor, tomaso de, authorized by the pope to obtain confessions through torture (shades of my sometimes dark sense of humor)

No matter about lack of familiarity with statistical algorithms - I've programed them into an excell spreadsheet - just input the handscored values (or measurements from Extract, Limestone, or hand-measurements), leave artifacted responses blank. it will handle up to five charts and up to 5 RQs, and will tollerate missing values.

it allows for the selection of pneumos (greater, weaker, combined, upper, or lower), and employs recognizable rank transformations (with those blasted tied ranks), standardizes values across component parameters, and can provide weighted mean component scores according to the desired contribution of components (EDA, cardio, pneumo, and PLE). The math is done automatically and instantaneously, and returns normalcy tests such as Levene's test for equality of variance and Quantile plots of normalcy, before calculating small sample t-tests for for mean differences of independent paired samples (CQ and RQs), along with one-sample t-tests for each RQ as a point estimate of the comparison mean. Because all of the requirements for parametric t-tests are not satisified (continuity of data, and intervalic data), I've also added a series of non-parametric tests, including a sign test (weak), and Wilcoxon-W (better). The Mann-Whitney test is used to calculate the RQ point estimates after re-ranking the dataset for each RQ (with the others ommitted). It is generally not advisable to conduct multiple Mann-Whitney test with more than two samples (which is what we have with multipel RQs and mixed issue tests), so I've included a Kruskal-Wallis H test (like a non-parametric ANOVA). Rather than evaluate each point estimate, Kruskal-Wallis tells us whether our separate RQs possess separate and distinct variances or distributions.

For interested persons, its quite possible to disect these equations and see how they work.

r

------------------
"Gentlemen, you can't fight in here, this is the war room."
--(from Dr. Strangelove, 1964)

posted 10-26-2006 04:39 PM            

I'll look for it. In general, the more I look at this, the more I'm underwhelmed. Its really not that difficult to understand or describe how this should work. However, it seems like we've maybe not done all our homework, and not fully integrated our procedures into the context of measurement sciences.

Its important to remember that any test score or measurement is not what scientists call a true-score or universe-score. For example, an IQ score is an estimate of a persons IQ. We do not pretend to assume that an IQ score represents someone's true IQ or universe score. We know that ALL test score are a combination of the universe (true) score and random measurment error - introduced by variability in physiology, psychology, component/sensor efficiency (sensitivity/specificity) and other sources of variability. So, our scores always have some variability. Using statistics, we can estimate the degree of efficiency that a test score accurately reflect the universe or true score - but we need to understand the variability to do so. Variability is in many ways dependent upon sample size, and for us that means the number of iterations of each test question, and the number of relevant and comparison questions as a whole. When we measure response to several iterations of a test questions we are not simply measuring the response to the test question, we are measure the variability of repsonse to that test question. Events like the Jack Ruby polygraph (7/18/64) in which 70+ relevant questions are nothing more than pretense that a single test measurement represents the universe score.

----------

OK, I got the manual. Interesting light reading.

hmm. they report a prior probability of .5 - it appears they are using a Bayesian evaluation at some point, probably as a final step. This would allow for the reasonable transformation of the test statistic from a probability of a certain result to a probability of deception (I've always wondered how they justified that). Do you know if polyscore does the same thing. I was thinking about this last weekend. Only using .5 is perhaps overly simplistic, and could be improved upon through other decision matrices.

I sure wish I had more time, or could get paid for this nonesense.

r

------------------
"Gentlemen, you can't fight in here, this is the war room."
--(from Dr. Strangelove, 1964)

posted 10-26-2006 10:58 PM            

Thanks for directing me to that Barry. Its somewhat similar to the 1988 study, but a little more accessible. (I've been meaning to send that to you but I still don't have a solution to a 32 meg attachment - perhaps you could get it from an .ftp space).

That is one of the more satisfying descriptions. However, I'm still courious how they (CPS/Kircher/Raskin) handle dimensionless standardization prior to the Bayesian calculation. The info on their webpage indicates it is based on verified criminal investigations.

I like the Bayesian caluculation idea. I've ranted to Lawson Hagler for a couple of years that we should do a study on base rates of various types of non-compliance admissions from sex offenders - pornography, masturbation fantasy problems, contacts with minors, unreported sexual contacts, etc. Those base rates could become a prior probabilities, that can be additionally weighted by the offender's risk level (even if in very general terms - low-moderate, moderate-high, Hare-psychopath). Quinsey et al, (2000) who developed the VRAG/SORAG have stratified the weight of varioug actuarially derived ricidivism risk prediction variables, for every 5% change in recidivism, up or down, from a base-rate of 35% - for violent and sexual offenders. Hanson and Bussierre (1996, 1998, and more recently) have a meta-analytic base rate of recidivism of %14 that is not stratified for risk level or diagnostic profile. Of course non-compliance rates are substantially higher than reoffense.

Anyway back to my yet inarticule questions about how Kircher and Raskin stratify dimensionless scores before the Bayesian analysis... How? Have they normed those? I don't think so. How then have they calculated the meaning of measured differences between relevant and comparison values. Norms would work, but require a sample. My attempt has been an ipsative evaluation of the significance of difference.

It seems clear that they have the same idea of z-scores for all CQs combined and individual RQs, but I think there is some missing description.

Evaluating the meaning of dimensionless scores requires a procedure typically dependent on norms or a distribution. I don't think its perfectly clear, but z-scores seem to create an implication of a simmetrical shape (like a normal distribution or t-distribution or binomial distribution).

However, the use of R/C ratios in OSS is different than Z-scores. Ratios - involving the division of two numbers both greater than Zero will create an asymetrical distribution shape - whose center is equal to 1, but will never be less than zero. Think about this the portion of this distribution that lies to the left of 1 will always fill the space from zero to 1, while the portion to the right of 1 can fill the space accross several numbers. This can be expected to produce an asymetrical distribution shape something perhaps like a chi-squared distribution. It would be interesting to see if log tranformation would transform this shape into something more normal, as sometimes happens. I don't think OSS bothers with fitting the distribution curve, but Krapohl and McManus (1999), and Dutton (2000) and Krapohl (2002?) provide normative tables derived from their assumedly representative development samples.

As I indicated previously, because a test score is always a component of the true/universe score and some variability, it is important to understand the distribution shape of that variability before we can determine how much of a difference in scored values becomes meaningful or informative.

The CPS stuff on Relative response magnitude is just another way of presenting dimensionless scores - by transforming observed z-scores to a scale from 0 to 100. Interesting, and graphical, but its still difficult to really appreciate the significance of the info.

I'd sure like to hear from people using CPS.

As far as the Polyscore critique goes, I think the simple version is that it may have been derived atheoretically, from an empirical approach towards feature selection. This is not really incorrect, but makes it difficult to articulate a theoretical explaination of why it works. Another

For another example, the famed MMPI (now MMPI-2 and MMPI-A) was developed atheoretically, with no attempt to explain why certain questions correlate with certain disorders - but the stats show they do. On the other hand Theodore Millon - whose work largely underlies our descriptions of Axis-II personality disorders in the DSM-IV-TR (along with some foundational work from the likes of Timothy Leary, during his pre-LSD days as a Harvard psychology professor) - designed the MCMI (now MCMI-III) from a theoretical framework. They do about the same thing in different ways, and they illustrate the importance of representative normative samples. The MMPI was normed on "normals" (loosely defined as people not housed at psychiatric hospitals), while the Millon Clinical Multiaxial Inventory was normed on forensic patients (and subsequently has a tendency to over-pathologize normal persons). With basic-normal neurotic mental health patients - who simply argue occasionally with others, but don't engage in serious criminal or deviant behavior - we don't use the MCMI.

This is the reason that the reviewers published on the Stoelting site, whom included John Kircher, griped so much about the lack of descriptives concerning the development sample.

Another known generalization flaw is that tests that involve larger numbers of features tend to overfit more readily, wherein they can become very descriptive of their development samples, but sometime don't generalize or cross-validate as efficiently as tests that involve fewer (and more robust) variables. A parallel to this is observed in reliability coefficients that are better, amoung human scorers, with fewer variables and poorer with more variables. While computers can offer excellent, if not perfect reliability with many variables, overfitting and cross-validation difficulties seem to be observed with computer-based atheoretical regressions, just as with theoretical models. One solution to this phenomena is to break large tests such as the MMPI (at about 576 questions) into smaller subscales. You can see the evolution of this in the work of Kircher and Raskin, who described 12 variables in 1988, and are down to three (the same as OSS) in the current CPS.

For a parallel example, look at recidivism risk for sexual offenders. Karl Hanson gave us one of the best models based on four (4) robust variables.

AFter all that digression, I've not articulated my remaining question. How has CPS evaluated the variability of test data, and what that tells us about the meaning of differences in test scores. They are clear about how they handled the similarity of data (through z-scores and mean z-scores). It seems like means and z-scores are primarily informative alongside their associated dispersion parameters. Maybe I missed, it with the quick read through, but I'm thinking I didn't see anything about how they calculated standard errors - which tells us a lot more about the actual meaning of a mean data value.

Perhaps the simlest illustration of the concern is that polyscore has been marketed as a proprietary black-box method, without a well-described and published description. Still, there is some evidence that it works, and there seems to be not a great deal of difference between the various scoring algorithms.

And now for the obvious... doesn't it seem obvious that if JHU/APL uses logistic regression to evaluate some 9,900+ possible features, chance alone will take over and return a few dozen aparently useful variables. Then - it gets better - the Extract program that seems to measure a handful of 8 or so features on behalf of the polyscore algorithm - depends so heavily on the very same few features described by Kircher and Raskin???

Alright, this has been a long ramble, and I have more questions than answers.

niters,

r

------------------
"Gentlemen, you can't fight in here, this is the war room."
--(from Dr. Strangelove, 1964)

posted 10-27-2006 09:40 AM            

I do think that our pneumo sensors are quite susceptible to noise/distortion from talking.

I was working on another matter, involving a world-class tennis athlete, and was surprised to observe all kinds of gadgetry that monitors some of the same physiology we monitor. Cardio (BP/heart rate), GSR, and respiration. Only, the athlete wears the sensors during the tennis match - it is apparently a lot less sensitive to noise. Data are recorded in a wristwatch, and synchronized to a video of the match. Perhaps that kind of equipment could better tollerate the interactive nature of your experiments.

Ahh, gadgets. I'm still waiting for a cell-phone in a shoe - someone has to have thought of that by now.

r

------------------
"Gentlemen, you can't fight in here, this is the war room."
--(from Dr. Strangelove, 1964)